Skip to content

[RNE Rewrite] feat: add voice activity detection pipeline#1298

Open
msluszniak wants to merge 16 commits into
rne-rewritefrom
@ms/rewrite-vad
Open

[RNE Rewrite] feat: add voice activity detection pipeline#1298
msluszniak wants to merge 16 commits into
rne-rewritefrom
@ms/rewrite-vad

Conversation

@msluszniak

@msluszniak msluszniak commented Jul 2, 2026

Copy link
Copy Markdown
Member

Description

Adds a Voice Activity Detection (VAD) task pipeline and a corresponding speech example app. Chunked inference, segment postprocessing and streaming run in TypeScript on top of the core model.execute primitive. The per-frame feature extraction (framing, mean-removal, pre-emphasis, Hann window) is a native speech.frameWaveform C++ op: on device it dominated a detect() call (~86%, ~40 ms of Hermes vs ~6 ms for the model forward pass), so per the extension guidelines it lives in C++. It writes straight into the pre-allocated model-input tensor, fusing mean-removal + pre-emphasis + Hann into one dependency-free (vectorizable) pass; framing drops to ~3.4 ms (~12×), below the model's own inference cost.

Introduces a breaking change?

  • Yes
  • No

Type of change

  • Bug fix (change which fixes an issue)
  • New feature (change which adds functionality)
  • Documentation update (improves or adds clarity to existing documentation)
  • Other (chores, tests, code style improvements etc.)

Tested on

  • iOS
  • Android

Testing instructions

Screenshots

Related issues

Closes #1249

Checklist

  • I have performed a self-review of my code
  • I have commented my code, particularly in hard-to-understand areas
  • I have updated the documentation accordingly
  • My changes generate no new warnings

Additional notes

  • Depends on the per-method get_dynamic_dims_forward input validation from the embeddings PR ([RNE Rewrite] Add text and image embeddings pipelines #1292): VAD feeds a variable-length [frames, 512] input tensor per chunk. Outputs are still validated exactly, so the output tensor is pre-allocated at the model-declared shape. Requires [RNE Rewrite] Add text and image embeddings pipelines #1292 to land. The fsmn-vad model is re-exported (tag v0.10.0) with a get_dynamic_dims_forward method returning int32 [rank, 3] bounds per input.
  • Segments are returned in seconds (the old native path returned raw sample indices).
  • The FSMN output contract is assumed to be [1, frames, classes] with class 0 = non-speech (speech = 1 - p0), matching the current native implementation.

@msluszniak msluszniak self-assigned this Jul 2, 2026
@msluszniak msluszniak added refactoring feature PRs that implement a new feature labels Jul 2, 2026
@msluszniak msluszniak linked an issue Jul 2, 2026 that may be closed by this pull request
@msluszniak
msluszniak force-pushed the @ms/rewrite-vad branch 2 times, most recently from f76a5c0 to ae3f0da Compare July 9, 2026 11:56
Port the VAD feature to the rewrite as a pure-TypeScript pipeline on top of
the core model.execute primitive (no new C++):

- src/extensions/speech/tasks/vad.ts: createVAD runner replicating the native
  FSMN-VAD algorithm (framing + Hann window + pre-emphasis, chunked inference,
  thresholding / min-duration / padding / merge). Segments are returned in
  seconds. Relies on the get_dynamic_dims relaxed input validation for the
  dynamic frame dimension; the fsmn-vad model is re-exported with it.
- src/extensions/speech/vadStreamer.ts: pure streaming state machine driving
  onSpeechBegin / onSpeechEnd over an accumulating buffer.
- src/hooks/useVAD.ts: hook wrapping createVAD + streamer lifecycle.
- Register models.vad.FSMN_VAD and export the speech extension.
- apps/speech: expo-router demo (mirrors apps/nlp) with a real-time mic VAD
  screen via react-native-audio-api.
…arams into model config

Frame geometry (sample rate, window/hop, FFT size, pre-emphasis, min frames)
is FSMN-specific and now lives on VADModel.featureConfig (supplied by the models
registry) instead of hardcoded constants in the task. The pipeline and streamer
are parameterized by it; detection thresholds remain generic VADOptions.
Framing (windowing, mean-removal, pre-emphasis, Hann) was ~86% of a detect()
call on device — ~40ms of Hermes vs ~6ms for the model forward pass. Port it to
a native `speech.frameWaveform` op that writes directly into the pre-allocated
model-input tensor, fusing mean-removal + pre-emphasis + Hann into a single
dependency-free (vectorizable) pass. Framing drops to ~3.4ms on device (~12x),
leaving it below the model's own inference cost.
The streamer re-ran the model over the whole growing buffer every tick
(up to ~8ms on a 10s buffer on device), even though only ~100ms is new
and the streaming decision only needs recent context. Cap the sliding
window to 2.5s so per-tick detect stays flat and low (~3x cheaper, ~2.7ms)
— still well above FSMN's receptive field and the 250ms min-speech-duration,
so detection is unaffected. On-device benchmark showed the alternative
'O(1) insert' buffer only saved ~6us/tick (0.2%), so it was dropped.
Fold the standalone vadStreamer into the task (createFsmnVad) as a pull-based
async generator woken by streamInsert, mirroring createWhisperSpeechToText:
stream()/streamInsert()/streamStop() replace the callback-based streamer.
Rename createVAD -> createFsmnVad and useVAD -> useFsmnVad for model-specific
naming, matching the Whisper STT pipeline so the two speech tasks share one
streaming structure.
@msluszniak
msluszniak marked this pull request as ready for review July 13, 2026 14:40
@barhanc
barhanc self-requested a review July 13, 2026 20:55
Comment thread packages/react-native-executorch/src/models.ts Outdated
…mments

Address review: the model config type is model-specific, so name it
FsmnVadModel (matching createFsmnVad/useFsmnVad). Also drop redundant
comments in fsmnVad.ts.
Comment thread packages/react-native-executorch/cpp/extensions/speech/operations.cpp Outdated
Address review: wrap the waveform/hann/dst buffers in std::span and take
per-frame subspans instead of raw float pointers; zero with std::ranges::fill
and sum the frame with a range-for. Trim the redundant Pass 1/Pass 2 comments.
Comment thread packages/react-native-executorch/src/extensions/speech/ops.ts Outdated
Comment thread packages/react-native-executorch/src/extensions/speech/ops.ts Outdated
Comment thread packages/react-native-executorch/src/extensions/speech/ops.ts Outdated
Comment thread packages/react-native-executorch/cpp/extensions/speech/operations.cpp Outdated
Comment thread packages/react-native-executorch/cpp/extensions/speech/operations.cpp Outdated
Comment thread packages/react-native-executorch/src/extensions/speech/tasks/fsmnVad.ts Outdated
Comment thread packages/react-native-executorch/src/extensions/speech/tasks/fsmnVad.ts Outdated
Comment thread packages/react-native-executorch/src/extensions/speech/tasks/fsmnVad.ts Outdated
Comment thread packages/react-native-executorch/src/extensions/speech/tasks/fsmnVad.ts Outdated
Comment thread packages/react-native-executorch/src/extensions/speech/tasks/fsmnVad.ts Outdated
- frameWaveform: validate tensor shapes via symbolic shapes in fromJs instead
  of manual rank checks; take startSample/numFrames/hopLength as uint64_t for
  automatic non-negativity checks; use asType<float> for preemphasis; add the
  missing waveform/hann pairwise aliasing check.
- speech extension barrel now exports ops (like cv), not the task.
- fsmnVad: drop the SampleSegment duplicate of Segment, inline
  Required<VADOptions>, and narrow the tInput try/finally to its actual use.
…ng, op API

- Replace the streaming generator with push()/resetStream(): a rolling window
  fed straight from a recorder callback, returning speechStart/speechEnd. The
  window is required — one 100ms chunk yields ~7 frames, far below the 25 hops
  minSpeechDurationMs needs, so detect() on a bare chunk never fires.
- Rename to createFsmnVoiceActivityDetector / useVoiceActivityDetection, mirroring
  createWhisperSpeechToText / useSpeechToText; frameWaveform -> extractFrames.
- Case acronyms per repo convention (cf. NmsOptions): VADOptions -> VadOptions etc.
- Drop VadFeatureConfig: model geometry is now file consts (fftLength read from
  model metadata) and the sample rate is exported as FSMN_VAD_SAMPLE_RATE_HZ,
  mirroring WHISPER_SAMPLE_RATE_HZ.
- extractFrames takes an options object and no startSample; the caller passes the
  per-chunk waveform slice, which removes the nested try.
- Fold scoresToSegments/mergeSegments into a single postprocess().
The FSMN model's output is dynamic, not just its input: N input frames
produce [1, N, classes], so execute() validates outputTensors against the
shape produced for the current chunk rather than the model-declared maximum.
The output tensor was pre-allocated once at that maximum, so every execute
threw 'outputTensors[0] must have shape [1, 247, 248] ... got 1000'. Inside
the recorder callback nothing surfaced the throw, so live detection silently
never fired.

Allocate the output per chunk with its frame dimension matched to the chunk
and dispose it alongside the other per-chunk tensors. Verified on a physical
Android device: live mic VAD toggles SPEAKING/SILENT again.
@msluszniak
msluszniak requested a review from barhanc July 15, 2026 12:09

@barhanc barhanc left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tested example app on Android and it works great. I don't have an iOS device to test unfortunately.

Comment thread packages/react-native-executorch/src/models.ts Outdated
Comment thread packages/react-native-executorch/src/hooks/useVoiceActivityDetection.ts Outdated
- FSMN_VAD_SAMPLE_RATE_HZ gets a @category Constants jsdoc; group the module
  constants together and add WINDOW_SAMPLES; rename DEFAULT_OPTIONS ->
  DEFAULT_VAD_OPTIONS.
- Output schema: the model always returns [1, F, C], so drop the redundant 2D
  alternative and the outFramesDim slicing; build the per-chunk output as
  [1, chunkFrames, numClass]. Fix the misleading 'variable rank' comment and the
  stale 'pre-allocates static output tensor' jsdoc. Rename chunkCapacity ->
  maxFrames.
- Remove the redundant waveform.length < FRAME_LENGTH guard (numFrames <= 0
  already covers it); rename push's speaking/isSpeaking -> isSpeaking/wasSpeaking.
- Trim comments/jsdoc per review (Hann, push, hook, operations.cpp).
- Inference method detect/detectWorklet -> detectVoice/detectVoiceWorklet
  (CV uses task-specific names: classify, detectObjects, transferStyle...).
- Hook useVoiceActivityDetection -> useVoiceActivityDetector (agent-noun,
  matching createFsmnVoiceActivityDetector and the other use* hooks).
- Registry key models.vad -> models.voiceActivityDetection (all other keys
  use full names; the 'vad' on-demand feature flag is unchanged).
- push -> detectVoiceOnStream (descriptive, pairs with detectVoice); its 'no
  transition' return is now VadEvent | undefined instead of | null.
- Move the extractFrames wrapper ops.ts -> extensions/speech/utils/vadUtils.ts.
  It's a model-specific op, not a domain-general one like cv's ops. Codify this
  in the add-native-extension skill: domain-general ops stay in ops.ts,
  model-/task-specific ops go under extensions/<domain>/utils/<name>.ts.
@msluszniak
msluszniak requested a review from barhanc July 16, 2026 16:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

feature PRs that implement a new feature refactoring

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[RNE Rewrite] Speech - add VAD pipeline implementation

2 participants